Skip to main content

ValkyrAI Workflow Engine

The ValkyrAI Workflow Engine is a powerful component that enables the creation, management, and execution of automated workflows. It provides a flexible and extensible framework for defining business processes, tasks, and execution modules that can be orchestrated to perform a wide range of operations.

Architecture​

The ValkyrAI Workflow Engine works via a Quartz scheduler that executes workflows on a schedule at intervals. Workflows can also be run and managed on demand via REST endpoints.

The architecture consists of the following key components:

  • Workflows: Define the overall process flow and contain a series of tasks
  • Tasks: Individual units of work within a workflow
  • Execution Modules: Specialized components that perform specific functions
  • Data Handlers: Components that process and transform data during workflow execution

When run by HeimdaLLM and via a collection of basic Execution Modules and in conjunction with OpenAPI Specs, the Workflow Engine is capable of nearly any task of any complexity.

Key Features​

Workflow Management​

  • Create, update, and delete workflows
  • Start, stop, and pause workflow execution
  • Monitor workflow status and progress
  • Schedule workflows using CRON expressions

Task Management​

  • Define tasks with specific execution parameters
  • Set task dependencies and execution order
  • Monitor task execution status
  • Handle task failures and retries

Execution Modules​

The Workflow Engine includes several built-in execution modules:

You can also create custom execution modules to extend the functionality of the Workflow Engine.

Workflow Definition​

Workflows are defined using a JSON-based configuration format that specifies the tasks, their dependencies, and execution parameters. Here's an example of a simple workflow definition:

{
  "name": "Document Processing Workflow",
  "description": "Process incoming documents and extract information",
  "schedule": "0 0/30 * * * ?",  // Run every 30 minutes
  "tasks": [
    {
      "id": "fetch-documents",
      "type": "files",
      "parameters": {
        "source": "/incoming-documents",
        "pattern": "*.pdf"
      }
    },
    {
      "id": "process-documents",
      "type": "document-processor",
      "dependsOn": ["fetch-documents"],
      "parameters": {
        "extractText": true,
        "extractMetadata": true
      }
    },
    {
      "id": "send-notification",
      "type": "email",
      "dependsOn": ["process-documents"],
      "parameters": {
        "to": "admin@example.com",
        "subject": "Document Processing Complete",
        "template": "document-processing-notification"
      }
    }
  ]
}

REST API​

The Workflow Engine exposes a comprehensive REST API for managing workflows, tasks, and execution modules. Here are some of the key endpoints:

  • GET /workflows: List all workflows
  • POST /workflows: Create a new workflow
  • GET /workflows/{id}: Get workflow details
  • PUT /workflows/{id}: Update a workflow
  • DELETE /workflows/{id}: Delete a workflow
  • POST /workflows/{id}/start: Start a workflow
  • POST /workflows/{id}/stop: Stop a workflow
  • GET /workflows/{id}/status: Get workflow status
  • GET /tasks: List all tasks
  • POST /tasks: Create a new task
  • GET /execution-modules: List available execution modules

Integration with Other ValkyrAI Components​

The Workflow Engine integrates seamlessly with other ValkyrAI components:

  • ThorAPI: Workflows can interact with APIs generated by ThorAPI
  • SecureFieldKMS: Sensitive data in workflows is protected using SecureField encryption
  • HeimdaLLM: AI-powered decision making and content generation within workflows
  • GridHeim: Data processing and spreadsheet operations

Getting Started​

To start using the ValkyrAI Workflow Engine, follow these steps:

  1. Ensure you have ValkyrAI properly installed and configured
  2. Create a workflow definition using the JSON format
  3. Use the REST API to create and manage your workflows
  4. Configure the necessary execution modules for your workflow tasks
  5. Start your workflow and monitor its execution

For more detailed information on specific execution modules, refer to the documentation for each module:

Best Practices​

  • Break down complex processes into smaller, manageable tasks
  • Use descriptive names for workflows and tasks
  • Set appropriate error handling and retry mechanisms
  • Monitor workflow execution regularly
  • Test workflows thoroughly before deploying to production
  • Document workflow configurations and dependencies